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Abstract 


This study investigated an alternative testing protocol used in an undergraduate managerial accounting 
course. Specifically, we assert that consistent open-book testing approaches will enhance learning and 
better prepare students for the real-world decision-making they will encounter. A semester-long testing 
protocol was executed incorporating a mix of open-book and closed-book pre-quizzes, and open-book 
major exams. Findings indicated that students taking open-book pre-quizzes performed better on open- 
book final exams, but not other major exams. Our research approach also revealed preliminary indications 
that our students value their textbooks more, and used them more frequently and extensively, to prepare 
for class using open-book testing protocols as opposed to using traditional closed-book testing procedures. 
Also, preliminary indications reveal that alternatives to traditional closed-book testing enhance student 
satisfaction with courses and textbooks, and provide the potential to improve students’ experiences in the 
workplace. We encouraged future quantitative studies with robust research designs dedicated to 
addressing these preliminary indications and provide several suggestions for future research. 
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Among educators, an excellent way to incite a debate is to discuss the strengths and weaknesses of any particular testing 
protocol. In an era of unprecedented change to the educational landscape, including curricula innovations, new 
approaches to teaching and assessment, and an emphasis on process improvement have created an environment that is 
often referred to as learning-centered (Ramaley & Leskes, 2002). Additionally, the proliferation of distance education 
and the abundance of credible on-line degree and certification programs have highlighted student interest and 
motivation. 

In this dynamic educational environment there has also been an increase in concerns about testing protocols, addressing 
learning outcomes, and assessment of student performance (Yang & Cornelious, 2005). These realities coupled with 
the inexorable transition from traditional pencil-and-paper exams to computer-mediated exams have authors 
investigating many different aspects of various new testing protocols including; test anxiety, preparation, and how 
students feel about various exam modes (Alltizer & Clausen, 2008). With alternative educational approaches, there are 
often concerns about cheating and plagiarism (Damast, 2007) and discussion on how to address them (Williams, 2006) 
continue as well. These pressures for transformation have also motivated educators to investigate a wide variety of 
improvement opportunities including testing, assessment, and assurance of learning. We feel examining open-book 
exams, and making the findings available, will encourage educators to investigate and identify opportunities for 
educational process improvement. 


Testing, Assessment and Assurance of Learning 









Approaches to assessment are as varied as the educators that use them, and the academic disciplines they represent. The 
act of “grading” was long-viewed as the best, or at least adequate, means of assessing students’ learning. As 
researchers consistently determined that tests and quizzes measure retention as well as knowledge (Roediger & 
Karpicke, 2006a), and as the educational landscape continued to change, performance measurement with alternative 
forms of valid assessment became the holy grail for educators. 

For many business schools, a major change occured with the passage of the Association to Advance Collegiate Schools 
of Business International’s (Association to Advance Collegiate Schools of Business International [AACSB 
International], 2007) standards and requirements for a more structured approach to assurance of learning and the 
outcomes assessment process (Anderson-Fletcher, 2005). These new AACSB standards forced a re-examination of 
educational processes at business schools and encouraged teachers to continuously improve the quality of education. 
At our institution, we are keenly interested in investigating improvements to our educational processes and encourage 
sharing results with the academic community. 

Our investigation of the relatively unorthodox assessment approach of using open-book exams is an excellent example 
of an attempt to enhance learning by continuously improving our educational processes in the spirit of the AACSB 
standards. Specifically, our study investigated whether consistent open-book testing would improve student 
performance on major exams and ultimately better prepare our students for real-world operational decision-making 
environments they will encounter. It also examined whether open-book testing would improve students’ overall 
satisfaction with courses, justification of required textbooks, and enhanced motivation to learn. 

Significance 

From a general education perspective, there is an on-going movement to help students become “intentional learners” 
who are capable of adapting to new environments, integrate disparate knowledge, and experience continuous learning 
throughout their lives (Ramaley & Leskes, 2002). For the last two decades, practically all stakeholders in the 
educational process have been demanding that the educational community constantly search for improvements to 
student learning and success (Barr & Tagg, 1995). Given the exponential growth in readily available knowledge, the 
assertion that technology is making information increasingly easy to access is also significant. This changing 
technology has had an important influence on pedagogy as students’ behavior transform and adapt to contemporary 
realities such as digital textbooks or e-textbooks (Weisberg, 2011). 

At our university, the United States Air Force Academy (USAFA), we feel the use of open-book exams may align better 
with our stated learning objectives and the technology that our students access. More importantly, we feel open-book 
exams will be more representative of the professional setting our students will encounter upon graduation. 
Investigation of the benefits of open-book exams will address in part the persistent call for relevance and “real world” 
application (Collett, 2000) in higher education. Our study addresses and compliments many academia-wide initiatives 
on this topic. 

Our premise is that traditional time-proven pedagogy, instructional activities, and assessment techniques might not be 
optimal approaches for all disciplines. Specifically we feel this may be true for our study involving the relatively 
structured discipline of managerial accounting. In a contemporary dynamic workplace, we feel decision-making is 
essentially an “open-book” activity where managers do not rely upon memorized information to act effectively. There 
are many professional bodies that agree with this premise. For example, in 2005 the National Association of 
Communication Systems Engineers (NACSE) call closed-book testing “archaic” and “not reflective of the real world” 
and subsequently changed its certification examinations and training to open-book testing (Sosbe, 2005, p. 4). 

For years the same growing void that we recognized in education and practice has also been identified in the accounting 
community (Albrecht & Sack, 2000), and has been extensively documented (Apostolou, Watson, Hassell, & Webber, 
2001), researched (Paisey & Paisey, 2004; Phillips & Phillips, 2007), and promulgated (Accounting Education Change 
Commission [AECC], 1990, 1992). Stakeholders, such as students and employers, have exasperated the so-called 
capabilities-gap, by demanding what they want from higher education and the realities of what universities can provide. 
As a result, formal outcome assessment of accounting programs has become increasingly significant as accreditation 


bodies require evidence of assurance of learning (AACSB, 2007). 

Similarly, we feel that the type of learning the undergraduates at our institution experience can improve if we attempt to 
replicate the modern dynamic workplace they will encounter after graduation. Therefore we contend that open-book 
exams would be a closer representation to what graduates would encounter “on the job” including for example being a 
pilot, program manager, or an accountant. Feller (1994) felt closed-book exams test what students can memorize while 
open-book exams better represent real-life situations where considerably more resources are available. Granted, a pilot 
needs to memorize certain emergency procedures. However, since each emergency a pilot might encounter is by 
definition unique, they must be able to assess the situation and adapt appropriately. Even if our graduates do not fly, to 
improve learning, we assert that consistent open-book quizzing and examination protocols will better prepare students 
for the real-world operational decision-making they will encounter. 

Open-book testing also addresses the discontent associated with textbook purchase for courses. Students get frustrated 
when they pay large sums of money for textbooks that are either sparingly used during a course, or find that success in 
the course is not dependent on their use of the textbook. Students spend billions of dollars each year on textbooks with 
legitimate complaints of too frequent revisions and needless bundling. There are reports that textbook prices tripled 
from 1986-2004 (US Government Accountability Office [GAO], 2005). Economic realities have undoubtedly forced 
some students to choose courses based on whether a textbook is required or even based on the cost of the textbook. 
Authors are addressing this textbook crisis by studying alternatives to required textbooks such as library reserves 
(Pollitz, Christie, & Middleton, 2009). Finally, the federal government has tried to alleviate some of the textbook cost 
burden by increasing direct aid and suggesting that textbook costs be tax deductible for eligible filers (Supiano, 2009). 
We argued that there are few better ways to illustrate the value of an expensive textbook than to allow students to 
reference it during open-book exams, and optimally retain it for future use. 

Finally, we believe there may be valuable insights into open-book testing protocols that many educators may have 
dismissed in the past. We feel this is especially true in education communities that culminate the learning experience 
with closed-book computer-based certification tests such at the Certified Public Accounting (CPA) and Certified 
Management Accounting (CMA) exams. However, even as educators embraced computer-based exams in their courses 
as improvements to their assessment portfolios, research indicated no significant difference in student performance on 
computer-based exams versus traditional paper-based tests (Anakwe, 2008). We feel by including open-book exams 
into their assurance of learning repertoire, whether they are computer-based or traditional paper-based, educators will 
likely enhance student learning while addressing the needs of future employers. 

Study Landscape 

Curriculum improvement is an integral part of the mission of our institution, USAFA. We are very circumspect as to 
how we select and how we approach our improvement efforts. We take great strides to ascertain that any study that 
directly involves students receives particular scrutiny. The design of this experiment ensured that our learner-focused 
institutional goals and objectives were not compromised, our assurance of learning and assessment processes were 
enhanced, and our students received equitable treatment regardless of the testing methodology. To accomplish this we 
offered all of the major exams, including the final exam, for every student in the course, in an open-book format. We 
used the pre-exam quizzes solely as the testing vehicle and alternated open-book and closed-book versions depending 
across sections of the course. Also, the pre-exam quizzes represented only 10% of their course grade ensuring that 
motivation for accomplishing the pre-exam quizzes existed, but the overall impact on the final grade was nominal. 

Review of the Literature 

Since assessment is not unique to any specific academic discipline, the review of the literature on open-book versus 
closed-book testing we chose was somewhat eclectic. We relied upon a broader education-oriented body of knowledge 
for relevant discussions of open-book and closed-book testing protocols. However, we feel this approach enhanced 
rather than diluted any finding or indications associated with the study. 

Numerous research efforts across several different academic disciplines have studied, measured and reported on the 
efficacy of open-book versus closed-book exams. A cross-disciplinary review of the literature appears to be 


inconclusive as to whether “better” learning occurs, varied on whether “better” preparation occurs, and consequently is 
diverse on which approach is superior. For example, in an introductory biology course, Moore and Jensen purported 
that open-book exams actually impede long-term learning (2007). For an introductory statistics course, Block (2012) 
discovered that in addition to a reduction in anxiety, the use of open-book exams increased student enjoyment while 
encouraging deeper student learning. Also, a new dimension was investigated in several psychology courses with the 
addition of “cheat-sheets,” or student produced notes that are available during an exam. Results showed that students 
performed slightly better on open-book exams versus closed-book exams, but for students that predicted they would do 
better with open-book versus cheat-sheets, the authors found no difference between the two groups (Gharib, Phillips, & 
Mathew, 2012). 

Open-Book versus Closed-Book Exams 

The closed-book exam is an established approach to assessment in higher-education. It is both widely accepted by 
educators and frequently used (Theophilides & Koutselini, 2000) and basically tests how well a student uses the 
knowledge they can recall with no additional material available for use on the exam. On the other hand, open-book 
exams allow students to consult textbooks, notes, and other course-related material during the exam. Some educators 
may consider open-book tests less conventional, but they have gained popularity across the entire spectrum of 
education including primary, secondary, and higher education (Baillie & Toohey, 1997; Eilertsen & Valdenno, 2000). 
Impediments to widespread adoption of open-book exams in studies include indications that students spend 
considerable time looking for answer instead of formulating their answers, and that open-book exams result in a 
reduction of preparation time in studying (e.g., Boniface, 1985; Rakes, 2008; Theophilides & Koutselini, 2000). Also, 
Agarwal, Karpicke, Kang, Roediger, and McDermott, (2008) reported mixed findings relative to long-term or delayed 
retention of material. This behavior is complicated by changes in students’ study-behavior based on their expectancy of 
an open-book exam and its impairment on long-term retention (Agarwal & Roediger, 2011). While our study did not 
address the issue of student expectancy, it is an important issue to address in future studies given its impact on student 
performance in other studies. 

The literature suggests that open-book exams may need new instructional techniques that address different cognitive 
processes and knowledge levels. For example, open-book exams might need designs that give students every 
opportunity to demonstrate their knowledge level and what they can accomplish in the time allotted. Feller, (1994) 
recognized that teachers will have to pay more attention to teaching the higher-level skills which includes 
conceptualization, problem solving, and reasoning. This is not a new dilemma by any means. Some of the earliest 
writings on the subject highlight issues with open-book exams including that they will likely reduce study by allowing 
students with a false sense of security that will allow them to “slide through” with minimum study (Kalish, 1958). 
Also, the advent of the various forms of personal computers, search engines, and other trappings of an information-rich 
classroom environment, have created changes in pedagogy. Improvements to this aspect of education represents 
another educational research opportunity including “open-book, open-web” (OBOW) testing protocols (Williams & 
Wong, 2009). 

In general, the relevant literature varies greatly in its orientation and can be categorized by measurement of student 
performance, assessment of student learning, and identification of various behavioral effects on students such as exam 
preparation and test anxiety. This study attempted to address each of these aspects of the open-book versus closed- 
book exam debate. 

Measurement of Student Performance 

As previously referenced, Kalish’s (1958) early investigations into the potential impact of open-book exams addressed 
the contention that the opportunity to look up material at its source should provide greater accuracy of response than 
depending upon memory. While this position was not specifically validated, the fact that open-book exams measured 
different abilities was verified and this encouraged future study (Kalish, 1958). 

Differences in student performance was also noted when the exam format changed. For example, students who took 
open-book exams the entire semester experienced significantly lower grades on closed-book final exams relative to 
those who took closed-book exams the entire semester (Moore & Jenson, 2007). In a more recent study, student 


judgements of comprehension were higher when students benefited from being able to use the open-book format 
(Ackerman & Leiser, 2014). 

Using examinations that were specifically designed to test critical thinking and higher-order skills, Ioannidou (1997) 
compared results of students taking open-book versus closed-book exams. She concluded that there was no significant 
difference in the scores of students taking open-book versus closed-book exams, and found that students that expect an 
open-book test might have less study motivation (Ioannidou, 1997). Other studies directly assert that student 
performance is actually worse on open-book exams (Boniface, 1985). 

Assessment of Student Learning 

While assessing learning is important, many feel that tests can do more. Exams can enhance learning while also 
improving long-term retention (Roediger & Karpicke, 2006b). The question becomes not if tests are beneficial, but test 
implementation. For example, Agarwal, Karpicke, Kang, Roediger, and McDermott (2008) found that open-book 
testing recorded better initial performance, but the benefit did not continue. Others have stated that closed-book final 
exams do not adequately measure deep conceptual understanding. Williams’ (2006) position is that closed-book final 
exams encourage “cramming" and “data dumps” and suggest that closed-book invigilated exams have become 
anachronisms. 

This phenomenon is described in the education literature as deep versus surface learning (Entwhistle, 1997). In general, 
we feel deep learning is best for contemporary students, and open-book testing has been identified as an excellent 
means to stimulate deep learning. However, even recognizing this deep learning versus surface learning perspective, 
researchers in the field of medical education found opposite results. Heijne-Penninga, Kuks, Hofman, and Cohen- 
Schotanus (2008) determined that closed-book tests stimulated deep learning more than open-book exams partially 
because students had more motivation to study for closed-book exams. In a related study, Heijne-Penninga, Kuks, 
Schonrock-Adema, Snijders, and Cohen-Schotanus (2008) suggested that by breaking the vast amount of medical 
information into core knowledge and backup knowledge, open-book testing complements closed-book testing and 
would be useful for assessment programs. 

Another educational philosophy referred to as Constructionist Learning, contends knowledge is created by the students’ 
learning activities, not necessarily transmitted by direct instruction. Constructionists argue that learning will occur 
only when the learner is actively engaged (Williams, 2006) and we feel that open-book testing enhances engagement. 
Constructivism focuses on knowledge construction, not knowledge reproduction (Herrington & Standen, 2000). We 
also feel this position supports open-book testing. 

Williams and Wong (2009) argue that open-book exams are more authentic and more constructively aligned with stated 
learning outcomes. Their position is that closed-book exams are anachronisms given the needs of a knowledge 
economy and the incompatibility with constructivist learning theory (Williams & Wong, 2009). We also feel open- 
book exams compliment this educational philosophy. 

Eilertsen and Valermo (2000) viewed open-book tests as a means to encourage thinking at higher cognitive levels and 
promote study and teaching methods. One of their preliminary findings was that open-book exams stimulate learning 
and noted that the test itself could be an arena for learning (Eilertsen & Valermo, 2000). 

The field of managerial accounting recognized the changing role of the practitioner that requires a new skill set and 
approach to problem-solving (Siegel & Sorenson, 1999). Educators in the accounting community can potentially 
benefit from considering findings from Albrecht and Sack (2000) that a rule-based memorization for certifying exams 
is inefficient and do not prepare students for the business world. In a landmark study performed by Albrecht & Sack 
(2000), one participating accounting educator stated: 

An accounting student needs to know that there are technical rules and regulations. He or she doesn’t need to be able to 
tell me what FAS 124 is. I don’t even know what FAS 124 is, but if I need to know it, I know where to get it. (p. 37)”. 

As higher education embraces on-line pedagogy, an additional question of how the internet supports learning and how 


teachers best assess learning looms. Some feel that learning is fundamentally a social process and as our culture and 
technology evolve, so must higher education. Preparing students to answer fact-based multiple-choice questions by 
rote memory is not adequately preparing them for future careers. The key is to develop instructional approaches that 
foster innovation, creativity, and independent thinking (Bruckman, 2002). 

Behavioral Effects on Students 

There is considerable discussion as to whether students’ grades are strongly associated with “good” academic behavior. 
Educator should engage in activities that promote good academic behaviors, but Moore and Jenson (2007) found 
indications of the opposite occur. They found that compared with student facing a closed-book exam, students with a 
scheduled open-book exam were less likely to attend class and help sessions, or submit extra credit assignments (Moore 
& Jenson, 2007). The results also indicated that students preparing for a closed-book exam tended to postpone their 
study until the end of the semester and focused on the memorization of material in the textbook (Moore & Jenson, 
2007). 

Theophilides and Koutselini (2000) found that students studying for open-book exams tended to review various sources 
and integrated the information they reviewed. Further, during the open-book exam, students worked creatively and 
“probed deeply” into the material (Theophilides & Koutselini, 2000). Phillips found that open-book exams improved 
study skills by constructing tests with contextual clues that helped students effectively identify correct answers in the 
text (Phillips, 2006). 

Open-book exams also compliment a learner-centered approach to education. For example, the reduction in the level of 
anxiety of an open-book exam, whether warranted or not, may be a result of more comprehensive exam preparation and 
more consistent learning environment with students avoiding “cramming” (Theophilides & Dionysiou, 1996; 
Theophilides & Koutselini, 2000). 

However, we also recognize additional complex behavioral issues and possible negative impacts of open-book exams 
that other authors have identified. For example, the use of open-book exams may require professors to ask questions on 
cognitive levels beyond recall including conceptualization, problem solving, and reasoning (Feller, 1994). It might also 
add an additional burden on the instructor since creating effective and valid open-book exams requires a professor to 
expertly create an open-book exam. Anecdotally, students may engage in a race to see how quickly they can find 
answers to an open-book exam, as opposed to a guessing game of what questions they will face on a closed-book exam 
that should be committed to rote memory. The perceptions of harder questions and second-guessing the instructor 
might create anxiety for some students. In any case, these behaviors do not create optimal learning environments, but 
when recognized, can be mitigated. 

Our study hopes to add value to the rich open-book versus closed-book exam debate in an effort to decrease the gap 
between the knowledge that our students obtain in courses, the skills they develop when taking our exams, and how 
they will eventually perfonn in an operational setting. 

The Study 

Our study investigated whether an open-book versus closed-book testing protocol significantly impacted students’ 
performance on major exams and their attitudes regarding the textbook and the course. Specifically, we hypothesized: 

HI: Students taking open-book pre-exam quizzes will perfomi better on open-book exams than students taking 
closed-book pre-exam quizzes. 

H2: Students taking open-book pre-exam quizzes will see clearer links between the textbook and course 
material and believe they learned more in the course than students taking closed-book pre-exam quizzes. 


Methodology 


Students in our undergraduate introductory managerial accounting course experienced a semester-long testing protocol 
incorporating either open-book or closed-book pre-exam quizzes to prepare for three major open-book exams during 
the semester. The final exam was also an open-book exam. This offering is a required course for all undergraduate 
Management Majors at our AACSB-International Accredited business program and represented one of the largest 
sample populations available. Second, as mentioned earlier, the accounting educational community has aggressively 
embraced alternative assessment studies such as ours and suggest improving introductory accounting courses with 
pedagogy that emphasized increased student involvement in the learning process (AECC, 1990; 1992). Third, the 
course had a robust set of learning objectives which could be utilized in future studies to measure students’ 
achievement of learning objectives between testing protocols. 

The total sample size consisted of 235 students across ten separate sections of the course taught by four instructors. 
One-half of each of the four instructors’ sections of students prepared for each of the three open-book major exams 
with two open-book pre-exam quizzes, while the other sections of students prepared for these major exams with closed- 
book pre-exam quizzes. Each instructor ensured that students’ grades would have no impact by the testing protocol they 
experienced, and that there was no advantage for being a student in either protocol. The students in the two testing 
protocols took similar pre-exam quizzes and major exams (i.e., similar conceptual questions with different numbers) 
and the same final exam. 

Upon completion of the course, all students completed a survey of questions investigating their attitudes toward open- 
book testing and its relationship to the course, its textbook and learning. The survey instrument was the same for all 
instructors. Students responded to each statement using a 5-point Likert-scale where 1 was strongly disagree and 5 was 
strongly agree. The statements examined whether students saw clear links between the materials covered in the 
textbook readings and class lectures, if the exams were closely related to the textbook, if students recommended the 
open-book testing approach, and if students’ instructors provided suggestions on how to effectively use the textbook. 
Additional questions surveyed students as to whether they spent more time working problems or exercises in the 
textbook, or more time preparing the textbook for use as reference during the exam to prepare for the open-book exams, 
and whether they felt they learned more or less using the open-book testing approach. Finally, the survey asked if they 
had a false sense of security in preparing for the open-book exams. 

Results 

We used PASW Statistics 18 to analyze our data (See Table 1). Our findings indicated that students who took open- 
book pre-exam quizzes did not perform significantly better on any of the three open-book major exams than students 
who took closed-book pre-exam quizzes. Flowever, students who took open-book pre-exam quizzes did perform 
significantly better on the open-book final exam than students who took closed-book pre-exam quizzes. As might be 
expected, students’ performance on the open-book final exam was significantly impacted by their performance on the 
three open-book major exams. Thus, these findings only partially supported our first hypothesis. 

In examining the survey data, we did find significant differences between students in the open-book versus closed-book 
pre-exam quiz sections. Specifically, students in the open-book sections more strongly agreed that they saw clear li nk s 
between materials covered in textbook readings and class lectures (M = 4.32 for open, 4.12 for closed) and the exams 
were closely related to textbook (M = 4.18 for open, 3.99 for closed) than students in the closed-book sections. The 
open-book students also more strongly agreed that their instructor provided suggestions on how to effectively use the 
textbook (M = 4.04 for open, 3.82 for closed). Although there was not a statistically significant difference between 
open-book and closed-book students and their recommendation of the open-book testing method, 82.6 percent of the 
students strongly agreed or agreed with the statement “I recommend the open textbook method for other classes (M = 
4.26)”. 

Although not statistically significant across our open-book versus closed-book testing protocol, 51.3 percent of students 
said they spent more time working problems/exercises in the textbook to prepare for the open-book exams, whereas 
48.7 percent of student said they spent more time preparing the textbook for use as reference during the exam. Eighty- 
eight percent of students felt they learned more using the open-book testing approach with no statistically significant 
difference between students in the open-book versus closed-book sections. We did find a statistically significant 
difference between students in the open-book and closed-book sections and their having a false sense of security in 


preparing for open-book exams. Specifically, although 70.5 percent of all students indicated they did not have a false 
sense of security in preparing for the open-book exams, students in the open-book sections had less of a false sense of 
security, as the mean for students in the open-book sections was 0.23 and 0.36 for students in the closed-book sections 
(where 0 was ‘no’ and 1 was ‘yes’). Thus, our students did not possess overconfidence in preparing for the open-book 
exams. 


Table 1. Means, Standard Deviations, and Correlations, 


Variable M SD 1 _ 2 _ 3 _ 4 _ 5 _ 6 7 8 _ 9 _ 10 

1. Open book 

quizzes; .49 .50 


0 = no. 
1 ~ycs 


2. Quiz 1 

8.71 

1.69 

.07 

- 


3. Quiz 2 

5.96 

2.09 

.05 

.06 

- 

4. Quiz 3 

6.77 

2.33 

.05 

.06 

.13* 

5. Quiz 4 

7.06 

2.32 


.IS*""' 


6. Quiz 5 

5.91 

2.39 

-.04 

.14*" 

] 4** 

7. Quiz 6 

7.08 

2.23 

.10 

.09 

.05 

S. Exam 1 

76.55 

10.15 

.01 

.26*** 

.05 

9. Exam 2 

75.69 

12.93 

-.03 

.26*“* 

.10 

10. Exam 3 

80.86 

10.83 

.01 

26*—* 

.15** 

11. Final 

194.31 

23.07 

12** 

20*** 

.15** 


Exam 


.11* 

.01 

.21*** 

-.01 

.09 

.17**- 

-.01 



.09 


.21“*“ 

2’>*K4«>4 


- 

.19*—* 

OS 

.21 — 

.07 

,40’*'*"'*' 

.40*"** - 

1 £*** 

2g*M<4* 


| -y «***»♦» 

.36*** 

.40**“* .46*“***** 


NOTE: N = 235. 

* p < .10. ** p < .05. *** p < .001. 


Findings and Discussion 

Although we found only partial support for our hypothesis that open-book pre-exam quizzes would significantly 
increase students’ performance on open-book major exams, we believe our study was successful in demonstrating 
usefulness of this testing protocol. Specifically, students like the idea of open-book exams, but not necessarily for the 
reasons educators might think. Anecdotally, students indicated they learned more through the open-book testing 
approach than they do through the conventional closed-book approach regardless of their grades on the assessments, 
because they were able to focus on mastering concepts to solve the accounting problems rather than memorize technical 
aspects they could look up in the textbook. Further, they also indicated they did not have a false sense of security going 
into the open-book exams, as they knew they would have to be able to work through the problems and apply concepts 
rather than simply report facts from the textbook, as well as having a time-constraint. These findings can provide a 
foundation for suggested open-book exam “best practices.” The key to successful implementation and increased 
student learning rests with how well the open-book testing protocol and process are described to the students. Also, 
best practices might include effort to ensure instructors are as unified as possible in their desire to investigate new 
approaches to enhancing student learning. 

Conclusion and Suggestions for Future Research 

In our quest to prepare our students for careers in the “open-book world” they will encounter after graduation, we 
believe open-book exam approach is useful for enhancing student learning while effectively preparing our students for 
real-world operational decision making. Even though our study did not answer the proposition of whether open-book 
testing encourages life-long learning in students by enlightening them that they do not need to “know” all the answers, 
we still feel that referring to source material for guidance is an attribute. There were indications that open-book testing 
encouraged active student engagement in their learning, and in many cases expanded their confidence in being able to 
work through difficult concepts. This as well as several other areas warrant further investigation, and we offer several 
suggestions for future study. 

First and foremost, it would be beneficial to conduct a follow-on study in which the open-book versus closed-book 
treatments are carried throughout the entire course with all the instructors. Additionally, even with over 250 students, 












in our opinion, we did not have enough subjects to warrant the multiple experimental treatments; open-book and 
closed-book pre-exam quizzes crossed with open-book and closed-book exams. Specifically, we recommend having 
one-fourth of the students in a large population course experience the following treatments: open-book pre-exam 
quizzes and open-book exams, closed-book pre-exam quizzes and open-book exams, open-book pre-exam quizzes and 
closed-book exams, and closed-book pre-exam quizzes and closed-book exams. We believe this approach would 
accommodate a more robust study and may produce significant findings. 

It would also be very interesting to conduct longitudinal studies of students at various levels of academic experience and 
development. These longitudinal studies offer the promise of understanding open-book preparation and performance 
differences for students with different levels of experience and academic maturity. Well-constructed longitudinal 
studies could also follow specific students throughout their academic careers to evaluate possible effects the open-book 
methodology has on student development and performance. These longitudinal studies can also give greater insight into 
possible negative impacts of open-book exams on student learning and performance. For example, mixed findings have 
been reported regarding long-term or delayed retention of material covered in open-book exam approaches (Agarwal, 
Karpicke, Kang, Roediger, & McDermott, 2008). Longitudinal studies are important as these long-term effects of the 
methodology are more carefully studied and reported. 

Many other potential research opportunities in the study of open-book exams exist. Educators agree that open-book 
exams reduce anxiety, and students appear to consistently prefer open-book exams despite the acknowledge 
understanding that they generally require the exhibition of higher-order thinking skills (Brightwell, Daniel, & Stewart, 
2004). However, as discussed earlier, questions might actually be harder on an open-book exam because they are 
testing at a higher level of learning. Consequently, we suggest pre-test post-test treatments of attitudes associated with 
this phenomenon. Also, although open-book exams do not necessarily lead to higher achievement in terms of test 
scores, they do seem to reduce unnecessary rote memorization of facts which allows students to prepare for exams 
more constructively (Theophilides & Dionysiou, 1996). We did not specifically address this phenomenon, but 
recognized it during our study and suggest future investigation. 

Open-book exams may also need new instructional techniques that address different cognitive processes and knowledge 
levels. In other words, perhaps studies should be conducted on the manner in which a student approaches a problem, 
based on the open-book versus closed-book experience. Related to this idea, open-book exams may be more difficult to 
construct (Eilertsen & Valdermo, 2000), and faculty may spend relatively too much time preparing and grading open- 
book exams while other faculty may also be insufficiently trained or experienced in open-book exam construction 
(Vanderburgh, 2005). We suggest additional research be oriented toward the instructor, not just the student. 

There are also numerous other hybrid options of open-book versus closed-book exam protocols; including an open 
hand-written note-card approach that show promise in not only high levels of learning but enjoyment of the course 
(Block, 2012). Research, such as the one conducted by Gharib, Phillips, & Mathew (2012) studying “cheat-sheets” 
versus open-book or closed-book exams, warrant further investigation. 

Finally, as discussed early, no open-book exam study would be complete without properly addressing advancements 
that are occurring in the on-line education environment. We feel it should not be the ubiquity of the internet that drives 
the exam approach; it should be a question of if students leam better via open-book exams that also allow access to the 
internet. We feel that perhaps the richest potential area of study may be to investigate the learning effects of not only 
open-book exams, but open-computer exams as well. To increase the real-world feel of an exam and to better replicate 
a future professional work environment, open-book open-computer exams, such as those suggested by Williams and 
Wong (2009), could be implemented and their impact on learning enhancement studied. 

Future research oriented toward addressing these potential research topics will contribute to the ongoing open-book 
versus closed-book exam debate. These educational process improvement efforts will help address demands that the 
educational community enhance student learning and success, as well as future professional performance. 

We are hopeful that educators in other disciplines find our study useful for their course development. At a minimum, 
we hope the readers of this study gained insight into a testing protocol they may have dismissed in the past, but now 
may consider adding to their assurance of learning repertoire in their quest to enhance their students’ learning and 


future professional success. 
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APPENDIX I 


Textbook Survey 

Instructions: 

In an attempt to enhance learning, we are very interested in your opinion about how we present course material. You 
are to read each statement and indicate your own personal feelings about the use of textbooks in this course by marking 
how much you agree with each of the following statements. Using the scale below, fill in the appropriate bubble on 
your General Answer Sheet. 

A B C D E 

Strongly Disagree Neutral Agree Strongly 
Disagree Agree 

1) I use textbooks to prep for exams in other classes in the same way I am using my textbook to prep for exams in my 
accounting class. 

2) My instructor provided suggestions on how to effectively use my text book. 

3) I use the textbook publisher’s website to supplement the textbook. 

4) I see clear links between materials covered in textbook readings and class lectures. 

5) Exams were closely related to the textbook. 

6) After receiving my score from a graded test, I reviewed the tested material again in the textbook. 

7) I recommend the open textbook method for other classes. 

8) I plan to sell my accounting textbook at the end of the semester. 

9) The price I paid for my accounting textbook was fair. 

10) The unremitting and nonsensical use of mildly amusing clipart in class lectures made it difficult to focus on 
learning objectives from the textbook. 

Please provide short answers (not just “yes” or “no” responses) to the following questions in the area provided. 

11) What was your strategy for using your textbook in your accounting class? 


12) Did you spend more time working on problems/exercises in the text or more time preparing the text for use as a 
reference during the exam? 



13) Do you feel you learned more or less using the open textbook approach? 


14) Did you have a false sense of security in preparing for the open-textbook exams? Please discuss why or why not. 


15) Please provide any other comments or observations you may have on the use of textbooks in this course. 


.^Corresponding author's email: steve.green@usafa.edu 
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